Patterns  and  Conflicts  for  the  Specification  and  Verification  of  Cognitive  Models 


F.  Mill,  A.  MacKlem 
Oakland  University, 
Rochester,  MI48309-4478 
mili@oakland.edu 


C.  Adams,  S.  Dungrani 


RDECOM/TARDEC 
Warren,  MI  48397-5000 


Abstract 


Cognitive  modeling  is  the  creation  of  computer-based  processes  that  mimic 
human  problem-solving  and  task  execution  using  existing  cognitive  theories. 
Cognitive  modeling  remains  a  labor-intensive  and  error  prone  activity  with  little 
theoretical  and  tool  support.  In  particular,  we  propose  an  approach  to  capturing 
specifications  for  cognitive  models  in  an  incremental  and  modular  way.  We  then 
discuss  ways  of  proving  that  that  a  cognitive  model  meets  its  specification. 

Introduction 

Cognitive  modeling  is  the  creation  of  computer-based  processes  that  mimic  human 
problem-solving.  Knowledge-based  cognitive  models  capture  task-specific  knowledge. 
They  are  built  to  run  on  cognitive  architectures,  which  are  virtual  machines  capturing 
general-purpose  regularities  in  human  cognition,  such  as  knowledge  acquisition 
(learning),  knowledge  use  (problem  solving),  and  knowledge  decay  (forgetfulness). 
Cognitive  architectures  are  an  embodiment  of  cognitive  theories.  The  most  notable 
cognitive  theories  and  associated  architectures  are  Soar  and  ACT-R.  They  both  originated 
at  Carnegie  Mellon:  SOAR  was  developed  by  Newell  et  ah;  ACT-R  was  developed  by 
Anderson  et  al.  The  two  architectures  have  many  common  aspects  and  components 
(cognitive  processor,  memories,  and  buffers)  and  some  differences  in  the  processes  by 
which  they  learn,  use,  and  forget  knowledge  and  in  the  processes  that  capture  various 
behavioral  moderators  such  as  fatigue,  fear,  and  other  emotional  factors.  Cognitive 
models  are  used  in  the  laboratories  for  experimental  research  in  cognitive  science  and  in 
industrial  applications  to  play  a  role  traditionally  played  by  a  human  (e.g.  automatic 
piloting),  for  simulation  and  training  (e.g.  war  gaming)  and  in  the  entertainment  industry 
to  create  virtual  actors,  and  credible  computer  game  characters.  Because  architectures  are 
by  design  low-level  virtual  machines,  cognitive  models  for  non-trivial  tasks  are  lengthy 
and  complex.  Models  created  to  perform  a  single  task  easily  exceed  thousands  of  rules. 
Because  the  state  of  the  art  in  the  software  engineering  of  cognitive  models  is  still  in  its 
infancy  stage,  models  are  typically  created  from  scratch;  a  model  created  for  one 
architecture  cannot  be  used  with  another;  the  validation  of  models  is  exclusively  done 
through  extensive  testing;  there  is  little  reuse  taking  place,  and  when  models  are  reused, 
the  process  of  adapting  and  combining  models  is  still  tentative,  manual,  and  ad  hoc.  One 
of  the  major  impediments  to  progress  on  the  above  aspects  is  the  absence  of  formal 
specifications  and  formal  definition  of  model  correctness.  In  this  paper,  we  focus  on 
specification  and  verification.  In  section  2,  we  propose  a  language  and  a  methodology  for 
writing  specifications  for  cognitive  models.  In  section  3,  we  define  the  semantics  of 
cognitive  models  in  terms  of  trace  languages.  In  section  4  we  formulate  the  problem  of 
model  correctness  as  a  comparison  between  context-free  languages.  In  section  5,  we 
show  a  heuristic  graph-based  verification  method.  We  use  the  towers  of  Hanoi  problem 
as  a  running  example. 
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2.  Specification  Methodology  and  Language 
2.1  Challenges  to  cognitive  models  specifications 

Although  specification  and  verification  of  traditional  sequential  programs  are  no  longer 
research  issues,  they  have  been  almost  absent  in  cognitive  models  in  particular,  and 
artificial  intelligence  programs  in  general.  Two  main  inhibitors  to  formal  specification 
and  verification  can  be  identified: 

1.  Tasks  for  which  cognitive  models  are  developed  are  ill-structured  and  hard  to 
specify  especially  that  their  requirement  is  that  they  perform  the  task  at  hand  in  a 
human-like  manner.  As  a  result,  cognitive  models  have  been  validated  mostly 
through  testing  by  comparing  their  traces  to  results  generated  from  human 
experimentation  [1,2,  11,  12,  14]  to  show  that  they  emulate  human  performance. 

2.  Because  cognitive  models  emulate  the  processes  of  human  problem  solving,  they 
cannot  be  adequately  captured  with  the  most  widely  used  specification  languages 
and  formalisms,  which  are  state-based  (precondition,  post-condition)  [6]. 

These  two  above  inhibitors,  while  real,  can  be  overcome  as  follows: 

1.  The  ill-structured  nature  of  the  tasks  can  be  addressed  by  using  a  specification 
formalism  that  allows  for  an  incremental  formulation  of  the  specification. 

2.  The  fact  that  cognitive  models’  mission  is  to  emulate  humans  performing  a  task 
reflects  different  components  of  the  requirements:  (i)  perform  the  task  (ii)  perform 
it  in  a  human-like  manner.  The  “human-like”  requirement  is  typically  qualified 
further  by  specifying  the  specific  aspects  for  which  the  model  must  be  similar  to  a 
human.  This  may  include:  decisions  made  and  actions  taken,  rationale  used  for  the 
decisions,  type  of  errors  made,  frequency  of  errors,  learning  taking  place,  timing 
characteristics,  task  switching,  etc.  Both  components  of  the  requirements:  (i)  and 
(ii)  need  to  be  captured  explicitly  before  hand. 

3.  Research  in  the  area  of  software  specifications  has  generated  a  wealth  of 
specification  languages  adapted  to  capturing  requirements  of  a  variety  of  systems 
including  history-based  systems  such  as  concurrent  processes  and  real  time 
systems.  Some  of  these  specification  formalisms  can  be  adapted  to  the 
specification  of  cognitive  models. 

In  other  words,  the  specification  of  cognitive  models  must  encompass: 

•  The  specification  of  functional  requirements,  i.e.  capture  the  requirement  that  the 
task  be  realized. 

•  The  specification  of  other  aspects  of  cognitive  behavior  such  as  timing,  errors, 
task  switching,  etc. 

The  specification  language  must  be  able  to: 

•  Capture  state-based  behavior  (pre-condition,  post-condition). 

•  Capture  history-based  behavior. 

The  specification  language  and  methodology  must  support: 

•  The  modular  specification  of  tasks  in  an  incremental  way. 


2.2  Running  Example 


We  use  the  example  of  the  Tower  of  Hanoi  throughout  this  paper.  In  this  problem,  there 
are  three  pegs  A,  B,  and  C  and  three  disks  of  three  different  sizes:  small,  medium,  and 
large  placed  on  the  pegs.  The  object  of  the  task  is  to  transfer  all  three  disks  from  peg  A 
to  peg  B  with  the  constraint  that  the  disks  must  be  moved  one  at  a  time  and  that  no  disk 
can  be  placed  on  top  of  a  smaller  one. 


Initial  Configuration 


Figure  1:  Towers  of  Hanoi 


A  state-based  specification  of  this  task  would  consist  of  a  transcription  of  the  information 
in  Figure  1.  Obviously,  this  infonnation  is  insufficient.  A  complete  specification  of  the 
task  would  include  the  definition  of  legal  moves  as  well  as  the  definition  of  the  specific 
aspects  that  need  to  be  human  like  and  how. 

2.3  Multi-layered  specification 

For  the  sake  of  separation  of  concerns  and  reusability,  we  recommend  distinguishing 
between  at  least  two  separate  layers  in  the  specification: 

•  Functional  layer:  in  which  we  capture  what  constitutes  a  valid  execution  of  the 
task  at  hand.  For  example,  the  functional  layer  specification  of  the  Tower  of 
Hanoi  specifies  that  disks  are  moved  one  at  a  time,  that  no  disk  is  placed  on  top  of 
a  smaller  one,  and  that  eventually  the  disks  are  all  placed  on  peg  B  in  their  final 
configuration. 

•  Cognitive- layers:  these  layers  capture  specific  aspects  of  the  trace  that  make  it 
human-like.  The  aspects  of  interest  vary  depending  on  the  underlying  architecture 
and  depending  on  the  application  at  hand.  The  underlying  architectures  enable  the 
mimicking  of  some  aspects  of  human  behavior;  for  example,  ACT-R  is  tuned  to 
duplicating  the  exact  timing  with  which  humans  make  decisions  at  the  50ms  level 
and  tuned  to  predicting  some  type  of  errors.  On  the  other  hand,  EPIC  focuses  on 
emulating  the  process  by  which  humans  interleave  the  processing  of  data  from 
different  sensors  (vision,  hearing,  touching)  and  motor  actions.  The  domain  of 
application  dictates  the  characteristics  that  are  of  particular  interest.  Emotional 
accuracy  maybe  more  critical  than  intelligent  behavior  to  make  a  computer  game 
character  compelling.  For  the  Tower  of  Hanoi  problem,  the  cognitive  layers 
would  capture  such  things  as:  the  occurrence  of  trial  and  error;  the  occurrence  of 
learning  by  not  repeating  the  same  mistake;  the  average  time  with  which  the  task 
is  executed. 

There  are  a  number  of  advantages  to  separating  the  different  layers  of  the  specification. 
On  the  one  hand,  this  allows  us  a  separation  of  concerns  and  facilitates  the  elicitation  of 


specifications.  On  the  other  hand,  by  capturing  the  functional  layer  separately,  the  same 
functional  specification  can  be  used  for  a  model  regardless  of  the  architecture  under 
which  it  is  being  implemented  and  of  the  application  in  which  it  is  used.  Similarly,  some 
aspects  of  the  cognitive  specification  can  be  captured  in  a  generic  way  and  reused  for 
different  tasks.  Another  benefit  of  the  separation  is  modularity.  Even  if  some  aspects  of 
the  cognitive  layer  can  be  hard  to  fonnalize,  we  can  at  least  capture  the  functional  level 
and  verify  the  model’s  functional  correctness.  The  other  aspects  can,  then,  be  left  for 
testing. 

2.5  Specification  Language 

The  specification  of  what  constitutes  a  valid  execution  of  a  task  can  be  though  of  as  a  set 
of  constraints  on  the  traces  generated  by  the  execution.  The  trace  of  an  execution  is  the 
sequence  of  observable  events  perceived  by  or  initiated  by  the  agent  (model)  executing 
the  task.  Some  constraints  are  best  defined  positively,  such  as  “every  pick-up-disk  must 
be  followed  by  a  put-down-disk”.  Some  constraints  are  best  defined  negatively  such  as 
“Must  not  place  disk  x  on  top  of  disk  y  where  y<x”.  Positive  constraints  are  labeled 
patterns;  negative  constraints  are  conflicts. 

Before  defining  the  specification  language  formally,  we  illustrate  it  by  showing  patterns 
and  conflicts  from  the  specification  of  the  Tower  of  Honoi: 

Functional  Specifications  of  the  Tower  of  Hanoi: 

Pattern  PI.  Eventually,  all  disks,  the  Large,  then  the  Medium,  then  the  Small  must 
be  placed  on  peg  B  in  that  order.  This  pattern  is  represented  by  the  finite  state 
automaton  below.  The  initial  state  labeled  with  an  arrow  head  represents  the 
beginning  of  the  pattern.  The  final  state  labeled  with  a  dot  inside  the  circle  represents 
the  (successful)  completion  of  the  pattern.  The  labels  of  the  arcs  (state  transitions) 
represent  the  event  that  triggers  the  transition. 


We  will  ignore  for  now  the  first  transition  labeled  with  the  empty  action,  X  and  the 
loop  that  follows  it.  The  pattern  starts  when  the  large  disk  is  placed  on  peg  B  taking 
us  to  state  Si.  From  Si,  three  events  can  take  place: 

•  The  medium  disk  is  placed  on  peg  B,  which  takes  us  to  the  next  state,  S2. 

•  The  large  disk  is  removed  from  peg  B,  which  takes  us  back  to  state  So. 

•  Any  other  move  which  leaves  us  looping  on  the  same  state  Si. 


The  final  state  is  reached  as  soon  as  the  small  disk  is  placed  on  peg  B  (on  top  of  the 
medium  and  the  large). 

Pattern  P2.  Every  <remove  disk  x  from  peg  _>  must  eventually  be  followed  by  a 
<place  disk  x  on  peg  _>.  This  pattern  ensures  that  no  disks  are  removed;  they  can 
only  be  moved.  If  we  want  to  insist  that  the  removal  of  a  disk  must  be  immediately 
followed  by  a  placement  of  the  same  disk,  we  need  to  disallow  all  other  actions  until 
it  is  placed.  This  is  the  subject  of  conflict  C2  below. 

Conflict  Cl.  Pattern  PI  ensures  that  eventually  disks  are  placed  on  peg  B.  Yet, 
pattern  PI  does  not  guarantee  that  when  the  model  stops  all  disks  are  still  on  peg  B. 
We  add  the  requirement  that  states  that  any  disk  movement  that  follows  that  point  is  a 
conflict. 


<,  any  move, 


As  can  be  seen  above,  patterns  and  conflicts  are  represented  in  the  same  way. 
Whereas  patterns  state  what  should  happen,  i.e.  we  want  to  reach  the  final  state; 
conflicts  state  what  should  not  happen,  i.e.  their  final  states  represent  conflicts  that 
should  not  be  reached. 

Conflict  C2.  After  <remove  disk  x  from  peg  _>,  any  operation  other  than  <place  disk 
x  on  peg  _>  represents  a  conflict.  This  conflicts  disallows  picking  up  2  or  more  disks 
before  placing  any  of  them  back  which  violates  the  rules  of  the  game. 

The  above  two  patterns  and  conflicts  are  a  representative  sample  of  the  functional 
requirements  of  the  task  Tower  of  Hanoi.  They  also  illustrate  the  following 
characteristics  of  this  specification  approach: 

•  Modularity:  Writing  a  specification  consists  of  adding  patterns  and  conflicts  until 
we  are  satisfied  that  we  have  captured  all  the  aspects  of  interest. 

•  Separation  of  Concerns:  Using  patterns  and  conflicts,  the  specifier  can  capture 
functional  requirements  and  cognitive  requirements  independently. 

•  Encompasses  State-based:  Pattern  PI  and  Conflict  Cl  together  capture  the 
requirement  that  when  the  task  is  done,  the  post-condition  of  having  the  three 
disks  in  the  correct  order  placed  on  peg  B.  This  illustrates  the  fact  that  this 


approach  allows  us  to  capture  state-based  requirements,  in  addition  to  history- 
based  requirements  (sequence  of  events). 

•  Methodological  support:  In  the  methodology  subsection  we  discuss 
methodological  support  for  this  activity. 

We  are  ready  now  to  define  the  specification  language. 

Definition,  Specification. 

A  task  specification  S  is  defined  by  the  definition  of: 

>  an  alphabet  SA, 

>  a  set  of  languages  (called  pattern  languages)  PLi,  PL2 ,  . .  .PLk  on  SA,  and 

>  a  set  of  languages  (called  conflict  languages)  CLi,  CL2 ,  . .  .CLi  on  SA, 

This  specification  defines  the  language  SL  on  alphabet  SA  where 

sl=  PLi  nPL2n  ...  PLk  ncLi  hcl2  ...ncu 


The  specification  alphabet  SA  is  the  set  of  “observable  events”  of  interest  to  the  specifier. 
For  example,  the  Specification  Alphabet  for  the  tower  of  Hanoi  is  SA={<REMOVE, 
disk,  peg>,  <PLACE,  disk,  peg>  |  disk  in  {small,  medium,  large}  and  peg  in  {A,B,C}}. 

The  traditional  approach  to  defining  a  language  is  by  defining  its  underlying  grammar.  A 
grammar  G  is  formally  defined  as  a  quadruplet  G=  <T,  N,  X,  P>  (Denning  et  al.  1 978) 
where:  T  is  the  alphabet  (finite  set  of  terminal  symbols);  N  is  a  finite  set  of  non¬ 
terminals;  X>  a  sentence  symbol  not  in  N  or  T  and  P  is  a  finite  set  of  productions  of  the 
form  a  — >-(3. 

When  the  productions  are  all  of  one  of  the  following  forms 
A  — >  w 

where  A  is  a  non-terminal  symbol  and  w  is  a  non-empty  string  from  NuT. 

X  — ►  W 

X-a. 

where  X  is  the  empty  string, 
the  language  is  qualified  as  context-free. 

The  relationship  between  the  grammar  and  the  associated  language  is  defined  as 
follows: 

Definition:  derivation 

Given  a  grammar  G,  a  production  a  — >P,  and  two  strings  co=  cp  a  \|/  and 
co’=  cp  P  \j/,  we  say  that  co’  is  immediately  derived  from  co  in  G.  This  is 
denoted  by  co  =>  co’.  When  coi,  co2,  ...conis  a  sequence  of  strings  such  that 
each  is  immediately  derived  from  the  predecessor,  we  say  that  con  is 
derivable  from  coi.  This  is  denoted  by  coi  =>  con. 

■ 

Definition:  Language  L(G) 

The  language  L(G)  generated  by  a  formal  grammar  G  is  the  set  of 
terminal  strings  derivable  from  X: 

L(G)=  { co|  X  co  } 


Pattern  Grammars 

The  interpretation  of  grammars  provided  above  is  the  traditional  one.  Grammars 
can  also  be  interpreted  as  defining  patterns.  For  example,  consider  the  grammar 
G  =<T,  N,  P,  X  >  where  T={read,  write},  N={A},  and  P  consists  of  the 
following  two  productions: 

X  — ►  read  A 
A  — >  write. 

The  traditional  interpretation  of  this  grammar  defines  the  language  consisting  of  a 
single  sentence:  “read  write”,  i.e.  L(G)={“read  write”}. 

By  contrast,  the  pattern  interpretation  of  this  grammar  defines  the  constraint  that 
every  read  must  eventually  be  followed  by  a  write.  The  following  sentences 
satisfy  the  above  pattern: 

1 .  read  write  read  write  read  write 

2.  display 

3.  write  write  write 

4.  read  read  read  write 

5.  open  read  write  read  write  close 

In  each  of  the  sentences  above,  every  read  is  eventually  followed  by  a  write.  Sentence  1 
illustrates  the  fact  that  a  pattern  may  occur  any  number  of  times  in  a  sentence  therefore 
the  beginning  (end)  of  a  pattern  is  not  necessarily  the  beginning  (end)  of  the  sentence. 
Sentences  2  and  3  illustrate  the  fact  that  a  pattern  can  be  vacuously  met;  the  sentences  do 
not  include  read  (prefix  of  the  pattern),  thus  vacuously  satisfy  the  pattern.  Sentence  4 
illustrates  the  fact  that  different  occurrences  of  a  pattern  can  overlap  within  the  same 
sentence  and  share  some  of  their  symbols;  the  pattern  here  appears  three  times  (3  reads) 
all  of  which  are  terminated  with  a  common  occurrence  of  the  symbol  write.  Sentences  2 
and  5  illustrate  the  fact  that  the  sentences’  alphabet  is  not  restricted  to  that  of  the  pattern, 
but  is  generally  a  superset  of  it. 

We  use  a  two-step  process  to  define  this  interpretation  of  grammars.  1.  The  grammar 
defines  a  pattern  (which  is  a  language  as  specified  in  previous  section).  2.  The  pattern  in 
turn  defines  a  language.  For  example,  given  the  pattern  grammar  “read  write”  above,  and 
the  alphabet  {“read”,  Write”},  it  defines  the  pattern  read- write  which  defines  the  set  of  all 
sentences  that  meet  the  pattern,  i.e.  (read+write+  u  write  )  . 

Definition,  Pattern  Grammar: 

A  Pattern  Grammar,  PG  is  defined  by  a  quadruplet  PG  =<T,  N,  P,  X  >  as 
defined  for  languages  in  general. 


Definition,  Pattern: 

The  language  P  generated  by  a  Pattern  Grammar  PG  is  called  a  Pattern. 

I 

Definition,  Pattern  prefix  occurrence,  complete  pattern  occurrence: 
Given  a  pattern  sentence  ps=  Si,  So,  ...  sp  and  given  a  sentence  co=  coi,  CO2, 

. . . ,  con,  sentence  co  is  said  to  contain  a  pattern  prefix  occurrence  (of  size 


j)  of  pattern  sentence  ps  at  position  i  if  there  exists  an  injective  mapping  f 
from  [l..j]  where  j<p  into  [i...N]  such  that 

•  f(l)=i  the  pattern  occurrence  starts  at  position  i, 

•  Vk:l..n:  C0f(k)  =  Sk  mapped  positions  contain  the  same  symbols. 

•  V  k,l:  1  ..n,  if  k>l,  f(k)>f(l).  The  symbols  of  ps  must  appear  in  w  in 
the  same  order  as  they  do  in  ps,  i.e.  the  mapping  f  must  be 
monotonous. 

•  V  k:2..n,  V  j :f(k-l)..f(k)-l  coj  ^Sk  the  mapped  position  must  be  the 
first  occurrence  of  Sk  in  co  after  the  occurrence  of  Sk-i- 

When  co  contains  a  pattern  prefix  occurrence  of  size  p,  we  say  that  co 
contains  a  complete  pattern  occurrence. 


Example  of  pattern  prefix  occurrence: 

Given  the  pattern  sentence:  ps=  369  and  the  sentence  co  =1234567867,  co 
contains  a  prefix  of  ps  at  position  3.  The  mapping  is  shown  in  the  bolded,  red 
symbols  in  co.  There  is  at  most  one  pattern  prefix  occurrence  starting  from  one 
given  position;  the  second  6  in  co  cannot  be  part  of  a  pattern  prefix  because  it  is 
not  the  first  occurrence  of  6  that  follows  3. 

In  the  definition  of  pattern  prefix  occurrence,  prefixes  have  a  size  of  at  least  one. 
There  are  cases  where  we  need  to  allow  a  prefix  of  size  zero.  Consider  again 
pattern  PI  for  the  tower  of  Hanoi.  If  the  pattern  were  to  start  at  state  So  (instead  of 
X  ),  the  pattern  would  state  that  “once  the  large  disk  is  placed  on  B,  eventually, 
the  medium  then  the  small,  must  also  be  placed  on  peg  B”.  In  other  words,  if  the 
large  disk  is  never  placed  on  peg  B,  the  requirement  is  irrelevant.  Because  the 
pattern  starts  at  X  with  initial  transition  X,  what  it  states  instead  is  “once  nothing 
(X),  eventually,  the  large,  then  the  medium,  then  the  small  disks  must  be  placed  on 
peg  B”.  In  other  words,  the  pattern  PI  must  always  be  met.  To  allow  these 
unconditional  patterns,  we  amend  the  definition  of  prefix  by  allowing  empty 
prefixes  for  patterns  whose  first  transition  is  a  X  transition. 

Definition,  Pattern  Lansuase: 

Given  a  Pattern  Grammar  PG  =<T,  N,  P,  X  A  and  an  alphabet  A  such  that 
TcA,we  define  the  pattern  language  PL(PG,A)={co|  coe  A  :  every 
prefix  occurrence  of  a  pattern  string  in  co  is  a  complete  pattern  occurrence 
of  a  pattern  string  (not  necessarily  the  same)  in  co.} 


Definition,  Compliance  with  a  pattern: 

A  language  L  with  alphabet  A  is  said  to  be  compliant  with  a  pattern 
defined  by  grammar  PG  iff  L  c;  PL  (PG,  A). 


Conflict  Grammars 


In  the  same  way  that  patterns  define  what  must  happen,  conflicts  are  used  to 
define  what  must  not  happen.  For  example,  the  grammar  G  =<T,  N,  P,  X  > 
Where  T=  {close,  read},  N={A},  and  P  consisting  of  the  following  two 
productions: 

X  — ►  remove  (f)  A 
A  — »  open  (f) 

The  language  defined  by  this  grammar  is  L(G)=  {“remove  (f)  ,  open  (f)”}. 

The  conflict  defined  here  is  that  once  an  object  (f)  is  removed,  it  cannot  be 
opened. 


Definition,  Conflict  Grammar: 

A  Conflict  Grammar,  CG  is  defined  by  a  quadruplet  CG  =<T,  N,  P,  £  >  as 
defined  for  languages  in  general. 

Definition,  Conflict: 

The  language  C  generated  by  a  Conflict  Grammar  CG  is  called  a  Conflict. 

■ 

Definition,  Conflict  Language: 

Given  a  Conflict  Grammar  CG  =<T,  N,  P,  X  >,  and  an  alphabet  A,TcA, 
we  call  the  conflict  language  CL(CG,A)={co|  coe  A  :  there  is  no  complete 
occurrence  of  any  conflict  sentence  in  co.} 


Definition,  Compliance  with  a  conflict: 

A  language  L  with  alphabet  A  is  said  to  be  compliant  with  a  conflict  iff  L 
c  CL  (CG,  A). 

We  revisit  the  definition  of  specification  given  in  the  beginning  of  this  chapter  by  stating 
how  the  pattern  and  conflict  languages  are  defined. 

Definition,  Specification  by  pattern  and  conflict  grammars. 

A  task  specification  S  is  defined  by  the  definition  of: 

>  an  alphabet  A, 

>  a  set  of  pattern  grammars  PGi,  PG2 ,  . .  .PGk  on  A,  and 

>  a  set  of  conflict  grammars  CGi,  CG2 ,  . .  .CGi  on  A, 

This  specification  defines  the  language  SL  on  alphabet  A: 

SL=  PL(PGi  ,  A)  fl  ...  PL(PGk,A)  D  CL(CGi  ,  A)  ...nCL(CGi,  A). 


2.6  Specification  Methodology. 

Capturing  the  right  specifications  is  at  the  same  time  critical  and  challenging.  The 
literature  in  software  specifications  is  rich  with  lists  of  qualities  that  specification 
processes  must  possess  and  that  software  specifications  must  have.  In  (Mili  et  al.  1994), 
we  capture  the  process  qualities  of  a  specification  by  two  properties:  completeness  and 
minimality.  A  specification  is  complete  if  it  captures  all  of  user’s  requirements.  A 


specification  is  minimal  if  it  captures  nothing  but  the  user’s  requirements.  Completeness 
and  minimality  cannot  be  formally  proven.  They  can  only  be  established  through 
redundancy.  We  define  a  software  specification  lifecycle  that  generates  redundancy  and 
uses  it  to  establish  the  completeness  and  minimality  of  a  specification.  We  organize  the 
specification  lifecycle  along  two  orthogonal  axes:  phases,  which  define  a  chronological 
structuring  of  the  process  (what  gets  done  when?);  and  activities,  which  define  an 
organizational  structuring  of  the  process  (who  does  what?).  We  identify  two  phases,  two 
activities.  The  partners  who  participate  in  the  lifecycle  are  the  user  group,  the  specifier 
group,  and  the  verification  and  validation  group.  The  two  activities  in  the  specification 
lifecycle  are: 

•  specification  generation.  This  activity  is  carried  out  by  the  specifier  group  with 
input  from  the  user  group.  It  consists  of  generating  the  specification  from  the  user 
concept,  possibly  adjusting  in  light  of  feedback  from  the  verification  and 
validation  group. 

•  Specification  validation.  This  activity  is  carried  out  by  the  verification  and 
validation  group.  It  consists  of  generating  redundant  requirements  information 
from  the  user  concept,  then  using  it  to  certify  the  generated  specification  or  to 
correct  it. 

The  two  phases  in  the  specification  lifecycle  are: 

•  Specification  generation.  During  this  phase,  both  the  specifier  group  and  the 
verification  and  validation  group  are  interacting  with  the  user  group  to  elicit 
requirements  from  it. 

•  Specification  Validation.  During  this  phase  the  verification  and  validation  group 
checks  whether  the  specification  derived  by  the  specifier  group  satisfies  the 
properties  generated  by  the  verification  and  validation  group.  Corrective  actions 
are  taken  accordingly. 

The  overall  process  is  summarized  in  Figure  4  below. 


Specification  Generation 
Activity 

Specification  validation  activity 

Specification 
Generation  phase 

Generating  specification 

Generating  redundant 
requirements  infonnation 

Specification 
Validation  phase 

Updating  the  specification  in 
light  of  V&V  feedback 

Matching  the  specification 
against  Validation  information. 

Figure  4.  The  Specification  Process. 


The  generic  specification  process  above  applies  for  the  specification  of  cognitive  models 
as  well,  with  each  of  the  activities  tailored.  We  discuss  them  in  turn: 

Generating  Specifications. 

As  stated  earlier,  the  specification  of  a  task  can  be  divided  into  function-level 
specification  and  cognitive-level  specifications.  The  process  described  here  can  be 
applied  to  each  of  the  levels  individually  or  to  all  of  them  combined. 


The  generation  of  the  specification  consists  of  generating  individual  patterns  and 
conflicts.  The  relative  independence  between  the  patterns  and  the  conflicts  allows  their 
generation  to  be  performed  independently.  The  process  of  generating  patterns  and 
conflicts  can  be  supported  by  tools.  We  are  currently  developing  a  tool  that  generates 
traces  consistent  with  patterns  and  conflicts  provided  by  the  specifier.  Traces  generated 
allow  the  specifier  to  tighten  the  specification  by  incrementally  adding  patterns  and 
conflicts  or  refining  the  ones  provided.  We  illustrate  this  with  the  scenario  shown  in 
Figure  5. 


Specifier:  enters  pattern  PI  starting  it  at  state  S0 
Tool:  generates  string 

<place  small  disk  on  peg  B> 

Explanation:  PI  met  because  not  triggered. 

Specifier:  edits  pattern  PI  by  adding  £  and  associated  transitions. 

Tool:  generates  string 

<place  large  disk  on  B,  place  medium  disk  on  B,  place  small  disk  on  B,  place  small  disk  on  C> 

Explanation:  PI  met  complete  occurrence  at  position  1 . 
Specifier:  adds  constraint  Cl . 

Tool:  generates  string  ... 

Figure  5:  Specification  Generation  support 


Generating  Redundant  Requirements  Information. 

The  role  of  the  Verification  and  Validation  group  is  to  capture  properties  that  can  be  used 
to  check  the  completeness  and  minimality  of  the  specification  generated  by  the  specifier 
group. 

The  verification  and  validation  group  focuses  on  generating  two  types  of  properties: 

1 .  Completeness  properties:  These  are  properties  that  the  verification  and  validation 
group  suspects  the  specifier  group  might  have  missed,  making  the  specification 
not  complete.  The  verification  and  validation  group  can  capture  such  information 
with  patterns  and  conflicts.  Because  these  properties  need  to  be  matched  with  the 
specifications,  it  is  best  to  keep  them  as  simple  as  possible.  Completeness 
properties  can  be  captured  through  negative  examples:  set  of  traces  that  should 
not  be  allowed  by  the  specification  generated.  If  the  specification  generated 
allows  these  traces,  then  it  is  incomplete  because  too  permissive.  Examples  of 
completeness  properties  for  the  Tower  of  Hanoi  are  shown  in  Figure  6. 


NE1:  <place  large  disk  on  peg  B,  place  medium  disk  on  peg  B,  place  small  disk  on  peg  B> 
Explanation:  No  disk  can  be  placed  before  it  is  removed. 

NE2:  <remove  large  disk,  remove  medium  disk,  place  large  disk  on  peg  B,  ...> 

Explanation:  Once  a  disk  is  removed,  it  must  be  placed  immediately. 

Figure  6:  Sample  Completeness  properties  (Negative  Examples) 


2.  Minimality  properties:  These  are  properties  that  the  verification  and  validation 
group  suspects  that  the  specifier  group  might  have  included  (when  they  should  not 
have)  making  the  specification  not  minimal.  The  minimality  properties  can  also 
best  be  captured  using  examples,  positive  in  this  case.  These  positive  examples 
must  be  allowed  by  the  specification  generated;  if  they  are  not,  this  would  indicate 
an  over-specification.  Examples  of  minimality  properties  are  shown  in  Figure  7. 


PEI :  <remove  small  disk,  place  small  disk  on  peg  B,  remove  small  disk,  . .  .> 
Explanation:  Trial  and  error  should  not  be  disallowed. 

Figure  7:  Sample  of  minimality  properties  (positive  examples) 


Matching  Specifications  against  redundant  information. 

During  the  validation  phase,  the  specification  is  matched  against  the  completeness 
and  minimality  properties.  If  the  specification  is  such  that  it  “rejects”  all  negative 
examples  and  “accepts”  all  positive  examples,  then  it  is  certified.  Otherwise,  it  is 
reviewed  and  revised  accordingly. 


3.  Semantics  of  cognitive  models 

We  consider  a  model  (program)  simulating  the  actions  of  an  agent  executing  a  task. 

When  the  model  is  run,  it  generates  a  trace  of  events  (received  from  the  environment)  and 
actions. 

We  call  MA,  for  model  alphabet,  the  set  of  events  and  actions  that  figure  in  the  trace. 
Definition,  Trace  Language: 

Given  a  model  M,  we  define  the  Trace  Language  TL(M)  as  the  set  of 
traces  generated  by  the  model.  The  set  of  symbols  (events  and  actions) 
occurring  in  the  traces  constitutes  the  Model’s  Alphabet  (MA). 


4.  Model  Correctness:  theoretical  formulation 

Definition,  Correctness: 


Given  a  model  M,  with  model  alphabet  MA,  and  a  specification  S,  with 
specification  alphabet  SA,  if  SA=MA,  the  model  M  is  said  to  be  correct 
with  respect  to  specification  S  if  and  only  if  ML  cz  SL. 


The  above  definition  holds  when  the  two  alphabets  are  identical.  This  condition  is  too 
restrictive  for  two  reasons: 

1.  Specifications  are  typically  concerned  with  only  one  subset  of  the  events  and 
actions  of  the  model.  For  example,  if  the  task  is  a  file  manipulation  task,  the 
specification  may  be  exclusively  concerned  with  operations  affecting  the  file 
integrity  and  the  correctness  of  the  results  (open,  close,  read,  write),  the  actual 
trace  of  the  model  is  likely  to  include  other  events  and  actions  as  well  such  as 
manipulation  and  use  of  the  data  read  and  written. 

2.  Specifications  are  often  captured  at  a  higher  level  of  abstraction  than  the  model’s 
operations.  For  example,  in  the  specification  of  the  Tower  of  Hanoi  we  think  of 
move-disk  as  an  atomic  operation;  in  fact  at  the  model  level,  this  action  may  be 
represented  by  the  sequence  “select  disk;  select  destination,  pick  up  disk,  place 
disk”.  Therefore  the  single  symbol  “move-disk”  in  the  specification  alphabet  is 
represented  by  the  set  {select  disk,  remove  disk,  select  peg,  place  disk  on  peg}  in 
the  model’s  alphabet. 

The  first  difference  can  be  addressed  easily  in  light  of  the  way  the  language  SL  is 
defined.  The  languages  PL;  and  CLj  are  defined  function  of  the  given  Patterns  and 
Conflicts,  and  the  alphabet  A,  superset  of  T.  Therefore,  it  suffices  to  add  the  missing 
symbols  to  A  in  order  to  get  PL;  and  CLj  and  thus  SL  defined  on  an  alphabet  that  contains 
all  symbols  needed. 

The  second  difference  requires  a  transformation  of  one  of  the  languages.  We  discuss  both 
transfonnations  in  turn: 

>  Transform  SL  defined  on  SA  into  SL’  defined  on  MA.  The  transformation 

consists  of: 

o  Mapping  each  SA  symbol  into  (a)  sequence  of  MA  symbols  (e.g.  move- 
disk-to-peg  mapped  to  pick-up-disk;  select-destination-peg;  drop-disk-on- 

peg- 

o  Substitute  every  occurrence  of  every  symbol  of  SA  in  SL  (or  its  grammar) 
by  each  one  of  the  corresponding  sequence(s)  from  MA. 

>  Transform  ML  defined  on  MA  into  ML’  defined  on  SA’.  The  transfonnation 
consists  of 

o  Mapping  each  SA  symbol  into  (a)  sequence  of  MA  symbols  (e.g.  move- 
disk-to-peg  mapped  to  pick-up-disk;  select-destination-peg;  drop-disk-on- 

peg- 

o  Substitute  every  sequence  of  MA  symbols  in  ML  that  has  a  mapping  into 
an  SL  symbol  into  its  corresponding  symbol. 

o  Leave  any  non  mapped  MA  symbols  as  is. 

This  transformation  is  more  difficult,  but  presents  the  advantage  of  raising  the 
level  of  abstraction  of  ML. 


From  this  point  we  will  assume  that  A  is  the  common  alphabet  to  the  specification  and 
the  model. 


Definition,  Verification: 

Given  a  model  M,  a  specification  S,  the  verification  of  correctness  of  M 
with  respect  to  S  is  the  proof  that  the  language  T  defined  by  the  traces  of 
M  is  a  subset  of  the  language  L  defined  by  S. 

I 

Proposition: 

To  prove  that  a  model  M  is  correct  with  respect  to  a  specification  S,  it  is 
sufficient  to  prove  that  the  language  T  generated  by  M  is 

>  Is  compliant  with  each  of  the  patterns  Pi  and 

>  Is  compliant  with  each  of  the  conflicts  Cj . 

Proof: 

This  proposition  is  a  direct  consequence 
specification  language,  and  compliance. 

Proposition: 

The  verification  of  a  model  with  respect 
this  chapter  is  an  undecidable  problem. 

Proof: 

Theorem  about  unsolvable  problems  for 
etal.  1978). 

The  fact  that  the  correctness  problem  is  undecidable  means  that  there  is  no  general 
algorithmic  solution  for  it.  In  the  next  section,  we  restrict  the  scope  and  seek  an 
approximate  semi  automatic  approach. 


of  the  definitions  of  correctness, 


to  a  specification  as  specified  in 


context-free  languages  (Denning 


5.  Model  Correctness:  heuristic  manual  approach 
5.1  Triggering  graphs 

For  models  written  as  production  systems,  which  is  the  case  of  interest  here,  we  can  think 
of  the  trace  of  a  model  as  the  sequence  of  productions  fired  during  one  execution.  While 
the  set  of  possible  sequences  of  productions  is  not  easily  accessible,  an  approximation 
(superset)  of  it  can  be  easily  generated  from  the  triggering  graph  defined  below. 

Given  a  set  of  productions  Pi,  P2,  •  •  .Pn  where  each  production  is  a  pair  <condition, 
action>,  we  say  that  production  P,  triggers  production  Pj  if  the  action  of  Pi  makes  the 
condition  of  Pj  true.  For  example,  consider  the  pair 
Pi=  if  goal  is  A 

then  (display  A  and  set  goal  =to  B)  and 
P2=  if  goal  is  B 

then  (display  B  and  set  goal  to  C) 


Because  of  the  presence  of  parameters  and  variables  in  the  productions  conditions  and 
actions,  the  triggering  of  one  production  by  another  is  history  and  context  dependent. 
Consider  for  example  the  following  three  productions: 

Production  PI :  Production  P2:  Production  P3: 


If  goal  is  place-disk 
And  step  is  start 
Then 

Set  goal  to  select-move 


If  goal  is  place-disk 
And  step  is  select-move 
and  disksize  is  2 
Then 

Set  goal  to  check-move 
Set  destination  to  peg  a 


If  goal  is  place-disk 
And  step  is  select-move 
And  disksize  is  2 
Then 

Set  goal  to  check-move 
Set  destination  to  peg  b 


This  example  illustrates  the  fact  that,  in  general,  we  cannot  state  with  certainty  whether 
the  execution  (firing)  of  a  production  will  necessarily  make  the  condition  of  another  true. 
We  are  generally  satisfied  with  the  possibility  that  this  might  be  the  case.  Furthermore, 
triggering  graphs  are  useful  to  the  extent  that  they  are  constructed  easily  and  provide  us 
with  useful  information.  In  practice,  the  determination  that  a  production  may  trigger 
another  is  based  on  only  one  or  a  few  variables  that  are  easily  accessible  and  take  discrete 
values.  In  the  example  above,  we  would,  for  instance  base  the  decision  on  the  variable 
goal  alone,  thus  concluding  that  PI  triggers  PI  and  P2. 

The  triggering  graph  of  the  set  of  productions  is  defined  as  follows: 

Definition,  Triggering  Graph: 

Given  a  set  of  productions  P=  {  Pi,  P2,  . .  .Pn  },  the  triggering  graph  of  P  is 
the  directed  labeled  graph  <V,  E  ->  A  >  where  V,  the  set  of  vertices  is  P 
the  set  of  productions  (i.e.  there  is  one  vertex  for  each  production),  and  E, 
the  set  of  directed  edges  consists  of  the  pair  (vi,  Vj)  for  which  production  P; 
triggers  Pj.  The  label  associated  with  edge  (vi,  Vj)  is  the  action  of 
production  P;. 


PI 

If  A  then  B  < 


P2:  if  B  then  ... 


P3:  if  B  then 


Each  production  is  a  vertex 
There  is  a  directed  edge 
from  Vi  to  Vj  if  and  only  if 
production  i  may  trigger 
production  j 

edges  are  labeled  with  the 
action  of  the  triggering  prod. 


We  will  assume  that  every  cognitive  model  admits  a  root  production,  i.e.  the  production 
that  will  always  be  executed  first,  not  triggered  by  any  other.  This  assumption  is  not 
restrictive  because  if  it  is  not  met,  we  can  always  add  a  production  that  triggers  all  others 
with  a  label  k. 

With  this  assumption,  the  triggered  graph  of  a  model  is  a  rooted  directed  graph. 


Definition,  Triggering  trace: 

Given  a  triggering  graph  with  labeling  alphabet  A,  each  path  on  the  graph 
defines  a  string  of  A  symbols:  the  sequence  of  labels  on  the  edges 
traversed.  The  triggering  trace  language  of  a  model  M  TTL(M,  G)  is  the 
set  of  strings  defined  by  all  paths  on  the  triggering  graph  G  of  M  that  start 
in  the  root  of  the  graph. 

■ 

Proposition,  TL(M)  subset  TTL(M,  G): 

Given  a  model  M,  a  triggering  graph  G  of  M,  every  trace  of  M  is  also  in 
TTL(M,G).  In  other  words,  TL(M)  c  TTL(M,  G). 


5.2  Finite  State  automata 

We  restrict  this  study  to  regular  grammars.  A  pattern  is  a  rooted  directed  labeled  graph. 

5.3  Correction  heuristics 

Formulate  axioms  concerning  patterns  and  conflicts  rel.  triggering  graph. 

If  pattern  graph  subset  of  triggering  graph,  then  pattern  met. 

Need  to  look  for  all  prefixes. 

If  conflict  graph  is  not  in  triggering  graph  then  conflict  is  met. 

5.4  Graph  matching 

6.  Summary,  Conclusion 
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